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The  NATO  Science  and  Technology  Organization 
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Synthetic  Environments  for  HSI  Application, 
Assessment,  and  Improvement 
(STO-TR-HFM-216) 

Executive  Summary 

This  report  describes  a  modeling  and  simulation  approach  to  experimentation  that  utilizes  models  and 
operators  to  measure  system  performance  under  targeted  key  perturbations.  This  approach,  called  Synthetic 
Environment  for  Assessment  (SEA),  is  a  new  way  to  use  simulation  for  conducting  trade-off  analyses  and 
for  exploring  very  complex  design  spaces.  SEA  has  the  potential  to  progress  systems  from  simply  sustaining 
incremental  improvements  in  favor  of  disruptive  innovation  -  exploring  radically  new  ideas  that  change  the 
rules  of  warfare  and  national  defence.  This  is  the  goal  of  SEA  at  its  highest  level. 

SEA  provides  a  workable  way  to  solve  the  practical  problems  involving  the  use  of  simulation  in  the 
capability  development  and  procurement  processes,  but  it  also  creates  opportunities  decision-makers  have 
never  had  before.  Calibrated  scenarios  from  one  lab  can  be  used  in  another,  resulting  in  data  that  can  be 
fairly  compared.  Researchers  can  have  realistic  test  environments  available  to  test  and  compare  theories  of 
human  performance  in  a  variety  of  disciplines  without  having  to  expend  precious  resources  learning  the 
domain  or  building  “throw  away”  simulations.  Both  researchers  and  acquisition  professionals  can  explore  a 
large  number  of  potential  solutions  to  hard  problems  using  trade-off  techniques  for  comparison. 
Most  importantly,  capability  development  and  the  rest  of  the  procurement  community  can  use  SEA  as  a 
communication  mechanism.  For  example,  if  capability  development  proposes  that  a  new  technology  for  the 
use  of  unmanned  vehicles  has  a  specific  benefit,  they  could  communicate  this  to  acquisition  through  realistic 
combat  scenarios  within  SEA. 

This  report  presents  a  conceptual  model  of  SEA  that  can  be  used  to  identify  the  key  issues  of  concern. 
Use-cases  were  provided  to  show  how  SEA  could  be  used  with  significant  benefit  to  its  user  groups. 
Member  Nation  SEA  activities  are  described,  as  were  their  usage  similarities  and  differences.  It  was  found 
that  SEA,  in  form  if  not  in  name,  is  everywhere  and  what  was  missing  was  a  unified  attempt  to  bridge  our 
terms  and  efforts  in  order  to  share  experiences,  technologies,  and  results.  Thus,  this  report  explored  the 
descriptions  of  ongoing  SEA  activities  to  derive  a  detailed  architecture  for  SEA  that  identifies  resources, 
models  and  systems,  input,  outputs,  and  user  communities  resulting  in  the  core  technology  areas  within  SEA 
and  the  unique  technical  barriers  that  need  to  be  crossed  to  realize  SEA  in  its  fullest  form. 
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Environnements  synthetiques  pour  P  application, 
revaluation  et  P amelioration  de 
l’integration  homme-systeme 
(STO-TR-HFM-216) 


Synthese 

Le  present  rapport  decrit  une  approche  d’ experimentation  qui  passe  par  la  modelisation  et  la  simulation  et 
utilise  des  modeles  et  des  operateurs  pour  mesurer  le  fonctionnement  des  systemes  dans  le  cadre  de 
perturbations  cles  bien  ciblees.  Cette  approche,  appelee  «  environnement  synthetique  devaluation  » 
(SEA,  Synthetic  Environment  for  Assessment),  est  une  nouvelle  maniere  d’utiliser  la  simulation  pour  realiser 
des  analyses  de  compromis  et  etudier  des  espaces  de  conception  tres  complexes.  Au  lieu  de  soutenir  les 
ameliorations  progressives  des  systemes,  le  SEA  pourrait  favoriser  1’ innovation,  en  etudiant  des  idees 
totalement  nouvelles  qui  changent  les  regies  de  la  guerre  et  de  la  defense  nationale.  Tel  est  l’objectif  du  SEA 
a  son  niveau  le  plus  eleve. 

Le  SEA  est  un  moyen  utile  de  resoudre  les  problemes  pratiques  impliquant  l’utilisation  de  la  simulation  dans 
les  processus  de  developpement  et  d’acquisition  des  capacites,  mais  il  cree  egalement  des  opportunity 
auxquelles  les  decideurs  n’avaient  jamais  eu  acces  jusqu’alors.  Les  scenarios  etalonnes  dans  un  laboratoire 
peuvent  etre  utilises  dans  un  autre,  ce  qui  produit  des  donnees  assez  faciles  a  comparer.  Les  chercheurs 
peuvent  disposer  d’ environnements  d’essai  realistes  pour  tester  et  comparer  les  theories  des  performances 
humaines  dans  differentes  disciplines  sans  avoir  a  utiliser  de  precieuses  ressources  pour  apprendre  le 
domaine  ou  construire  des  simulations  a  usage  unique.  Les  chercheurs  et  les  professionnels  de  1’ acquisition 
peuvent  etudier  un  grand  nombre  de  solutions  potentielles  a  des  problemes  ardus  en  les  comparant  a  l’aide  de 
techniques  de  compromis.  Le  plus  important  est  que  la  communaute  de  developpement  des  capacites  et  le 
reste  de  la  communaute  d’acquisition  peuvent  employer  le  SEA  comme  mecanisme  de  communication. 
Par  exemple,  si  la  communaute  de  developpement  des  capacites  avance  qu’une  nouvelle  technologie 
d’utilisation  des  vehicules  sans  pilote  presente  un  avantage  particulier,  elle  peut  le  faire  savoir  a  la 
communaute  d’acquisition  au  moyen  de  scenarios  de  combat  realistes  au  sein  du  SEA. 

Le  present  rapport  expose  un  modele  conceptuel  de  SEA  qui  peut  servir  a  identifier  les  questions  cles. 
Des  cas  d’utilisation  ont  ete  foumis  pour  montrer  comment  utiliser  le  SEA  pour  le  plus  grand  benefice  de  ses 
groupes  d’utilisateurs.  Les  activites  de  SEA  des  pays  membres  sont  decrites,  ainsi  que  leurs  ressemblances  et 
leurs  differences  d’utilisation.  11  a  ete  conclu  que  dans  la  pratique,  meme  s’il  n’est  pas  designe  sous  ce  nom, 
le  SEA  est  present  partout  et  qu’une  tentative  unifiee  de  rapprochement  du  vocabulaire  et  des  travaux  faisait 
defaut  pour  partager  les  experiences,  les  technologies  et  les  resultats.  Ce  rapport  a  par  consequent  etudie  la 
description  des  activites  de  SEA  en  cours  afin  d’en  deduire  une  architecture  detaillee  de  SEA  qui  identifie  les 
ressources,  modeles  et  systemes,  entrees,  sorties  et  communautes  d’utilisateurs,  ce  qui  indique  les  domaines 
technologiques  centraux  du  SEA  et  les  obstacles  techniques  uniques  que  nous  devons  surmonter  pour 
realiser  pleinement  le  SEA. 
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SYNTHETIC  ENVIRONMENTS  FOR  HSI  APPLICATION, 
ASSESSMENT,  AND  IMPROVEMENT 

1.0  INTRODUCTION 

As  long  as  there  have  been  models  and  simulations  capable  of  reproducing  complex  task  environments,  there 
have  been  efforts  to  use  synthetic  environments  for  studying  system  performance. 

The  reasons  for  the  use  of  synthetic  environments  vary  across  a  number  of  critical  user  groups.  We  view  SEA  as 
the  confluence  of  software,  hardware,  and  context  from  users  in  a  variety  of  roles  (See  Figure  1 ).  The  software 
component  deals  with  models  and  simulations.  The  hardware  component  deals  with  computing,  physical 
interfaces,  and  equipment.  The  human  component  deals  with  the  different  roles  that  people  have  in  system 
design  and  the  organisational  responsibilities  they  represent.  The  NATO  Modeling  and  Simulation  (M&S) 
Master  Plan  (2012)  identifies  its  stakeholders  as: 

•  Support  to  Operations,  which  is  responsible  for  decision-makers  having  access  to  capabilities  required 
to  decide  on,  initiate,  sustain,  and  successfully  conclude  operations  (e.g.,  organisations  that  define 
strategy,  planning,  and  executing  operations). 

•  Capability  Development,  which  is  responsible  for  the  future  preparation  to  foster  continuous 
improvement  of  military  capabilities  in  order  to  enhance  the  interoperability  and  effectiveness 
(e.g.,  organisations  that  perform  doctrine  validation,  operational  analysis  in  support  of  operational 
requirements  definition  and  collection;  research  and  technology;  and  concept  development  and 
experimentation). 

•  Mission  Rehearsal,  which  is  responsible  for  the  preparation  and  rehearsal  for  a  planned  mission  or 
course  of  action  to  reduce  risk  and  surprise  and  to  improve  the  knowledge  and  awareness  of  situations. 

•  Training  and  Education,  which  is  responsible  for  collective  training,  individual  education,  exercises  and 
training  events. 

•  Procurement,  which  is  responsible  for  the  support  of  total  lifecycle  management  of  assets  and  systems 
including  design  risk  reduction,  test  and  evaluation. 


Figure  1:  SEA  as  the  Confluence  of  Human,  Software,  and  Hardware  Components. 
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Since  every  partner  country  has  their  own  way  of  assigning  roles  and  responsibilities,  this  paper  will  utilize  the 
NATO  Modeling  and  Simulation  (M&S)  Master  Plan  (2012)  which  identifies  its  stakeholders  as: 

•  Customers  that  determine  the  operational  needs  for  M&S  capabilities; 

•  Users  that  demand  M&S  capabilities; 

•  Suppliers  that  develop  M&S  solutions  or  provide  research  in  M&S; 

•  Coordinators  that  provide  consistency  between  stakeholder  groups;  and 

•  Advisors  that  advise  customers,  users,  and  suppliers. 

The  NATO  stakeholder  groups  are  represented  in  most  of  the  groups  listed  above  as  shown  in  Table  1. 


Table  1:  Mapping  NATO  M&S  Stakeholders  to  Organisational  Responsibilities. 


Support  to 
Operations 

Capability 

Development 

Mission 

Rehearsal 

Training  and 
Education 

Procurement 

Customers 

X 

X 

X 

X 

X 

Users 

X 

X 

X 

X 

X 

Suppliers 

X 

X 

Coordinators 

X 

X 

X 

Advisors 

X 

X 

X 

X 

X 

Over  time,  the  roles  of  the  human,  software,  and  hardware  will  change.  Figure  1  suggests  that  they  are  equally 
important  but  this  is  unlikely  to  be  the  case,  and  it  certainly  will  change  throughout  the  lifecycle  of  any  process. 
The  role  of  the  human  is  typically  the  most  critical  role  (excepting  perhaps  a  completely  autonomous  system) 
with  software  and  hardware  supporting  the  human  in  a  variety  of  ways. 

Researchers  are  interested  in  studying  behavioral  phenomena  and  testing  theory.  Acquisition  professionals  are 
interested  in  conducting  cost-benefit  analyses  and  comparing  trade-offs  against  one  another.  Analysts  might  be 
interested  in  big-picture  doctrine  and  policy  issues.  In  all  cases,  there  is  a  careful  balance  of  realism  against  other 
factors  in  deciding  what  simulation  to  use  and  how  to  use  it. 

Thus  far,  this  has  been  done  in  an  ad  hoc  fashion  with  little  thought  to  what  we  are  balancing  or  how  the 
different  competing  needs  might  be  harnessed  to  produce  reusable  synthetic  environments  for  assessment  that  fit 
more  than  one  user  group.  Reusability  is  critical  for  two  important  reasons.  First,  reusability  usually  results  in 
reduced  development  costs  and  reduced  cost  of  ownership  which  increases  opportunities  for  use  over  time. 
Second,  reusability  results  in  reduced  variability  in  assessment  because  the  reused  models  are  reliable 
“constants”  that  allow  increased  confidence  in  measuring  dependent  variables.  This  report  will  explore  these 
user  groups,  their  applications,  how  they  are  similar  and  different,  what  our  Alliance  partners  are  doing  that  is 
related  to  these  objectives,  and  how  we  might  pursue  the  use  of  Synthetic  Environments  for  Assessment  (SEA) 
on  a  grand  scale  that  could  benefit  the  international  community. 

Nations  represented  on  RTG  HFM-216  have  been  working  in  the  general  area  of  SEA  for  many  years, 
but  seldom  in  such  a  close,  coordinated  fashion.  Our  goal  was  to  explore  how  SEA  is  being  used  by  our  member 
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countries,  what  methods  they  use,  what  applications  spaces  are  being  exploited,  and  how  we  might  take  the  next 
steps  towards  a  coordinated  multi-country  demonstration  of  SEA  and  its  value  to  acquisition,  science  and 
technology,  training,  and  operations. 

1.1  Organisation  of  this  Report 

This  report  will  begin  with  a  discussion  of  Synthetic  Environments  for  Assessment,  what  SEA  is  and  what  it  is 
not,  and  how  SEA  compares  to  other  related  concepts.  From  this  high-level  description,  we  will  present  a  coarse 
architecture  of  what  SEA  should  be  in  terms  of  components  and  capabilities.  We  will  then  describe  potential 
use-cases  and  ways  to  measure  the  utility  of  SEA.  Next,  the  report  will  describe  a  number  of  SEA-like  activities 
being  pursued  by  our  RTG  Member  Nations.  We  will  highlight  the  similarities  between  these  programs  in  an 
effort  to  derive  a  detailed  architecture  with  technical  requirements.  This  will  allow  us  to  link  SEA  to  two  other 
NATO  activities  that  have  much  to  contribute  to  the  end  goals  of  SEA.  We  conclude  with  a  summary  and 
recommendations  for  future  NATO  efforts  in  this  area. 


2.0  SYNTHETIC  ENVIRONMENTS  FOR  ASSESSMENT 

2.1  What  is  SEA? 

The  operational  environment  is  becoming  increasingly  complex.  While  automation  continues  to  be  an  important 
part  of  the  solution,  the  role  of  human  operators  has  never  been  more  important.  What  we  seek  is  a  synergy  of 
manpower  and  equipment  resulting  in  optimal  performance  and  readiness  -  but  the  complexity  of  the  operational 
environment  limits  our  ability  to  predict  second  or  third  order  effects,  and  how  our  design  decisions  will  impact 
performance.  Current  practice  relies  heavily  on  corporate  knowledge,  experience,  physical  prototypes, 
and  intuition,  resulting  in  a  slow-paced  evolutionary  progress.  Leap-ahead  capability  often  eludes  us  because 
incremental  improvements  in  system  design  are  like  hill  climbing;  every  iteration  is  a  little  better  than  the  last, 
but  a  limited  perspective  makes  one  unable  to  see  the  much  larger  hill  nearby.  It  could  also  be  argued  that  this 
evolutionary  approach  limits  the  ability  for  materiel  developers  to  be  strategic.  The  argument  that  we  continue  to 
fight  the  last  war  holds  for  our  acquisition  practices  as  well  as  our  strategic  thinking.  SEA  is  a  framework  for  the 
development  of  tools  supporting  non-linear  innovation.  Put  another  way,  SEA  is  meant  to  facilitate  disruption 
and  disruptive  ideas  by  providing  a  new  way  to  explore  unconventional  designs.  SEA  is  not  a  product, 
but  rather  an  approach  with  a  corresponding  architecture  to  support  a  large  ecosystem  of  components. 

Warfighter  needs  are  addressed  by: 

1)  Requirements; 

2)  Science  and  Technology  (S&T); 

3)  Materiel  development;  and 

4)  Procurement. 

Each  of  these  steps  is  meant  to  blend  one  into  the  other,  leading  to  a  validated  and  effective  solution.  Elowever, 
requirements  are  often  vague,  poorly  expressed,  and  sometimes  are  written  to  an  already  assumed  “solution”. 
In  these  cases,  the  requirement  is  written  to  a  product  rather  than  to  a  capability  or  a  desired  outcome.  Capability 
development  is  not  needed  in  every  warfighter  solution,  but  in  cases  where  a  technology  gap  exists  to  meet  a 
desired  goal,  it  is  their  responsibility  to  develop  new  products  or  knowledge  to  fill  that  gap  so  that  a  materiel 
solution  can  be  developed.  But  S&T  is  often  not  warfighter  scenario  driven.  Certainly,  basic  science  should  be 
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given  a  pass  on  knowing  exactly  how  it  fits  into  future  defence  products,  but  as  S&T  becomes  more  applied  and 
closer  to  transition  to  the  materiel  developer,  it  should  be  driven  by  valid  warfighter  scenarios  and  performance 
metrics  that  will  tell  us  how  good  of  a  solution  it  is.  Furthermore,  procurement  can  often  dictate  the  selected 
solution  path  regardless  of  the  capability  required  or  the  “leap  ahead”  innovations  available.  Consequently, 
what  happens  too  often  is  that  S&T  develops  new  concepts,  processes,  and  capabilities  that  can’t  be  procured  or 
adopted  to  meet  future  needs. 

What  is  needed  is  a  way  to  close  the  gaps  between  these  four  critical  groups.  If  S&T  coidd  look  ahead  to  new 
potential  conflicts,  force  structures,  and  constraints  in  a  way  that  provided  direct  input  to  requirements, 
our  ability  to  think  and  act  strategically  would  be  greatly  enhanced.  If  S&T  had  calibrated  scenarios  and 
performance  metrics  that  were  developed  and  validated  by  the  warfighter,  they  could  use  these  to  explore  trade¬ 
offs  in  the  design  space  that  would  result  in  more  useful  transitions  to  the  acquisition  community.  Science  and 
technology  also  needs  a  better  way  to  communicate  ideas,  concepts,  and  proposed  solutions  to  the  materiel 
developer  and  the  warfighter  that  are  believable  and  testable.  Otherwise,  S&T  has  a  tendency  to  oversell  and  the 
materiel  developer,  consequently,  has  a  tendency  to  under  believe.  We  need  to  close  this  gap  so  that  S&T  is 
consistently  exploring  ideas  that  are  operationally  credible  and  materially  achievable. 

Similarly,  procurement  needs  calibrated  scenarios  and  tools  to  assess  competing  designs  throughout  the 
procurement  cycle,  not  only  in  the  early  phases  when  design  decision  are  being  made.  Procurement  also  needs 
better  ways  to  mitigate  risk  in  their  processes.  This  is  achieved  through  continuous  assessment  of  design 
alternatives,  similar  to  what  we  believe  SEA  can  provide.  Training  and  education  will  look  to  evaluate  trade-offs 
between  training  and  other  ways  to  improve  overall  system  performance.  They  will  also  want  to  predict  the 
training  value  of  a  proposed  training  system  as  well  as  assess  the  design  of  training  assets.  Lastly,  Test  and 
Evaluation  (T&E)  is  about  measurement  and  assessment.  SEA  provides  inexpensive  alternatives  to  T&E 
processes  and  should  add  value  to  live  fire  T&E  because  they  can  do  more  within  SEA  than  they  ever  could 
using  other  types  of  simulation.  In  all  of  these  cases,  and  for  all  of  these  user  groups,  we  see  SEA  as  a  valuable 
alternative  to  “business  as  usual”  that  mitigates  risk,  explores  more  of  the  design  space,  and  provide  substantive 
data  to  decision-makers  at  the  appropriate  time. 

2.2  A  Working  Definition 

Our  working  definition  of  SEA  is: 


A  modeling  and  simulation  approach  to  assessing  human-system 
performance  and  trade-offs  in  representative  mission  scenarios. 


SEA  allows  a  radical  expansion  in  our  ability  to  assess  new  alternative  designs  in  the  infinite  design  search 
space.  Throughout  the  system  acquisition  cycle,  SEA  facilitates  an  expansion  (as  new  ideas  arise  or  new 
requirements  are  identified)  and  pruning  (as  decisions  are  made  or  priorities  are  set)  of  the  design  space.  A  list  of 
trade-off  domains  could  be  presented  in  terms  of  DOTMLPF: 


•  Doctrine :  Can  the  problem  be  solved  by  a  change  in  doctrine  alone?  Use  SEA  to  explore  changes  in 
doctrine  with  measurable  outcomes. 

•  Organisation :  Can  the  problem  be  solved  by  changing  the  organisational  structure?  Use  SEA  to  explore 
how  changes  in  organisational  structure  impact  the  same  measurable  outcomes. 

•  Training :  Can  we  train  our  people  better  as  a  way  to  solve  the  problem?  Use  SEA  to  explore  if  improved 
human  performance  will  resolve  the  problem. 
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•  Materiel :  Can  the  problem  be  solved  by  an  acquisition  -  a  new  device,  platform,  or  improvement? 
This  could  be  a  better  interface,  automation,  computation,  improved  sensors,  or  communications. 
Use  SEA  to  insert  these  into  operational  scenarios  and  measure  the  same  variable  outcomes. 

•  Leadership  and  Education'.  Can  the  problem  be  solved  by  improved  leadership  or  education 
(new  knowledge)?  Probably  the  most  abstract  use-case  for  SEA,  but  we  could  explore  cause  and  effect 
of  education  on  personnel,  then  apply  it  to  an  operational  scenario  provided  by  SEA. 

•  Personnel'.  Can  the  problem  be  solved  by  adding  people  or  changing  the  people  available?  Use  SEA  to 
measure  the  effect  of  personnel  changes  on  the  operational  scenario. 

•  Facilities :  Can  the  problem  be  solved  by  adding  or  altering  facilities?  This  can  apply  to  materiel 
(facilities  to  acquire  new  systems)  or  training  (new  training  facilities).  Use  SEA  to  determine  if  these 
improvements  achieve  measurable  results  that  solve  the  problem. 

All  of  these  involve  cost,  but  to  optimize  investment,  we  need  to  better  understand  how  trade-offs  between  these 
competing  designs  impact  overall  system  performance. 

For  many  domains  of  interest,  capability  requirements  shift  far  too  fast  to  be  addressed  adequately  by  the 
conventional  “research-design-build-test”  cycle  of  development.  For  example,  we  might  wish  to  know  what  the 
impact  might  be  of  a  new  capability  on  training,  or  team  performance,  or  retention.  When  faced  with  a  new 
unanticipated  mission,  how  do  we  determine  what  new  capabilities  are  required  to  meet  published  performance 
standards? 

It  may  appear  that  these  are  the  same  questions  that  Simulation-Based  Acquisition  (SBA)  was  intended  to 
address.  SBA  is  described  as  a  process  enabled  by  robust,  collaborative  use  of  simulation  technology  that  is 
integrated  across  the  phases  of  the  acquisition  lifecycle.  It  is  not  specific  to  what  technologies  are  used  or  how 
those  technologies  are  connected.  The  “vision”  of  SBA  was  specifically  to  leverage  simulation  technology  to 
improve  the  acquisition  process. 

SBA  achieved  limited  success  for  a  variety  of  reasons,  most  important  among  these  being  a  failure  to  implement 
interoperability  standards  to  meet  specific  SBA  goals  (as  opposed  to  other  modeling  and  simulation  goals). 
SBA  was  successful  in  programs  that  had  the  capacity  to  develop  their  own  systems  to  address  their  specific 
concerns.  Smaller  programs  found  little  that  could  be  reused  or  repurposed  for  their  use. 

SEA  is  unique  in  that  it  is  an  architecture  that  allows  for  interoperable  models  to  be  reconfigured  in  infinitely 
many  ways  to  test  any  hypothesis  or  evaluate  any  design  feature.  Trade-off  analysis  should  be  far  more  reliable  if 
models  can  be  reused  so  that  factors  being  held  constant  are  really  held  constant. 

SEA  is  a  new  approach  that  does  not  demand  new  technology.  Little,  if  anything,  is  ground  breaking  in  SEA  as  a 
new  technology.  What  is  new  is  the  way  SEA  seeks  to  unify  efforts  through  strategic  architectures  that  encourage 
collaboration  and  reuse  of  data,  software,  models,  metrics,  scenarios,  and  analyses.  This  report  will  document 
several  ways  in  which  our  international  partners  are  accomplishing  this. 

2.3  SEA:  A  New  Approach 

SEA  provides  a  cost-effective  approach  that  focuses  on  reuse  and  validation.  In  Figure  2,  the  user  input  is  really 
a  need  statement  or  question.  These  needs  likely  relate  in  some  way  to  the  DOTMLPF  trade  space  we  discussed 
in  the  previous  section,  but  any  question  involving  design  to  improve  system  performance  would  be  relevant  and 
appropriate.  The  user  must  also  specify  or  characterize  what  a  “good”  solution  looks  like  (assessment  criteria). 
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How  is  it  to  be  measured?  This  can  be  as  vague  as  a  heuristic,  or  as  specific  as  a  numeric  measurable  outcome  - 
but  it  must  be  applied  to  outcomes  for  feedback  to  the  decision-maker. 


Figure  2:  A  Schematic  Diagram  of  SEA. 


Inside  the  SEA  “box”  we  observe  that  there  are  four  key  elements  of  concern  that  work  together  in  concert  in  a 
very  flexible  way.  The  two  most  flexible  elements  of  SEA  are  software  and  scenarios: 

•  Software  represents  the  independent  modules  that  fit  together  to  address  a  specific  question  or  set  of 
questions.  Modularity  and  the  use  of  data  standards  and  protocols  ensure  maximal  reuse.  Software 
modules  are  interchangeable  with  validated  scenarios.  Figure  3  lists  some  of  the  types  of  software 
modules  that  SEA  may  contain. 

•  Scenarios  represent  a  wide  array  of  validated  configurations  that  stress  the  appropriate  points  needed  to 
produce  results  that  the  decision-maker  can  use.  We  want  to  examine  the  boundary  conditions  to 
determine  where  improvements  can  be  found  and  what  the  costs  are  to  develop  those.  Scenarios  identify 
what  software  modules  are  needed.  Figure  3  lists  some  of  the  variables  that  an  SEA  scenario  might 
specify. 
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Figure  3:  Elements  of  Software  and  Scenario. 

SEA  is  not  meant  to  be  a  closed-loop  system.  It  is  intended  to  incorporate  humans  in  the  loop  as  well  as  actual 
hardware  systems  and  surrogate  hardware  mock-ups: 

•  Human  represents  individuals  or  teams  of  people  interacting  with  the  system.  They  may  be  operators  of 
equipment,  role  players,  instructors,  or  evaluators. 

•  Hardware  represents  actual  systems  or  mock-ups  meant  to  “stand  in”  for  actual  systems  where 
necessary. 

SEA  requires  standardized  approaches  to  make  all  these  parts  fit  together  easily  and  in  a  predictable  way. 

The  outcome  is  some  measure  of  success  as  defined  by  the  user  input.  These  data  points  can  be  visualized  or 
communicated  in  whatever  way  best  helps  the  decision-maker  to  understand  the  trade  space  around  the  questions 
that  we  initially  asked.  We  compare  the  output  to  what  was  described  initially  as  a  “good”  solution. 

2.4  A  Conceptual  Architecture 

At  a  conceptual  level,  and  without  yet  describing  the  detail  of  how  SEA  could  be  implemented,  we  refer  back  to 
Figure  2  above.  Requirements  from  each  of  the  domains  (customers,  users,  suppliers,  coordinators,  or  advisors 
within  acquisition,  training,  or  operations)  drive  the  specifications  of  what  the  synthetic  environment  must 
support.  The  purpose  of  the  SEA  is  to  allow  for  data  driven  trade-off  analyses  in  each  of  the  areas  described 
earlier.  This  is  where  science  and  technology  becomes  important;  these  trade-offs  and  the  implications  of  design 
decisions  are  at  the  heart  of  what  S&T  contributes  to  the  enterprise. 
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As  we  attempt  to  take  the  next  step  and  define  what  the  essential  parts  are  that  comprise  SEA,  we  need  to  look 
through  each  of  the  primary  domains  as  a  lens.  Each  domain  asks  unique  questions  and  demands  unique 
answers.  Expanding  what  we  discussed  in  Figure  2,  we  now  focus  on  three  primary  elements  (see  Figure  4): 


•  Scenario  defines  the  inputs  to  the  SEA  that  are  enabled  by  the  software.  These  are  very  different 
depending  on  what  domain  this  is  and  what  questions  are  being  asked  (see  Figure  3). 

•  Metrics  defines  the  outputs  from  the  SEA.  These  are  also  dependent  on  the  domain,  but  there  is  greater 
overlap  here  since  the  SEA  is  usually  measuring  some  aspect  of  performance. 

•  Method  is  the  way  the  SEA  links  inputs  to  outputs.  The  technical  solution  lives  here.  These  are  all  the 
individual  technologies  (many  from  the  M&S  discipline)  that  are  aggregated  in  unique  ways  to  answer 
specific  questions. 
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Figure  4:  A  Conceptual  Breakdown  of  SEA  as  Defined  by  Each  of  the  Primary  Domains. 
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The  procurement  community  is  interested  in  technology  or  assets.  Their  scenarios  would  include  different  types 
of  equipment,  weapon  systems,  sensors,  and  communications.  They  may  also  experiment  with  capabilities  that 
may  not  yet  exist.  For  example,  “What  is  the  impact  on  effectiveness  if  this  sensor  had  10%  greater  range?” 
They  may  also  study  new  configurations  of  existing  equipment  to  determine  if  a  new  acquisition  is  even 
required.  Here,  metrics  include  measures  of  performance  and  risk  reduction.  Use-cases  might  include  building 
confidence  in  radical  designs  through  iterative  test  and  evaluation,  modular  decomposition  with  test  and 
evaluation,  and  cost-benefit  analysis  of  competing  designs. 

Support  of  operations  is  interested  in  manpower  issues.  These  too  are  addressed  by  SEA.  Support  of  operations 
experimentation  might  involve  altered  manpower  models,  changing  training  requirements,  and  staffing 
procedures  while  measuring  readiness  as  a  dependent  variable.  Use-cases  might  include  validating  benchmarks 
with  metrics,  and  blurring  the  distinction  between  training  and  operations,  especially  if  SEA  efficiently 
encompasses  live  training. 

Test  and  evaluation  examples  abound  within  SEA.  Almost  everything  SEA  does  has  some  component  of  test  and 
evaluation  embedded  in  it.  If  we  consider  the  use  of  simulation  as  a  precursor  to  live  fire  test  and  evaluation, 
we  might  use  SEA  to  study  the  boundary  conditions  of  the  flight  parameters  of  a  proposed  missile  system, 
for  example.  SEA  woidd  also  be  used  to  “pre-test”  live  fire  exercises  to  ensure  that  the  operational  plan  and 
metrics  will  yield  a  result  that  answers  the  operational  questions  it  was  meant  to  address. 

As  a  support  function,  science  and  technology  touches  all  aspects  of  the  other  domains,  but  adds  a  number  of 
distinct  requirements  specific  to  hypothesis  testing,  discovery,  and  transition  to  application.  It  is  desirable  to  have 
“portable”  SEAs  with  selectable  fidelity,  so  that  a  facility  in  one  location  that  might  include  high-end  simulators, 
live  equipment,  and  real  operators,  can  be  shared  with  another  location  that  has  only  a  network  of  laptops. 
SEA  accomplishes  this  via  its  modular  approach  that  separates  each  component  into  individual  parts  that  may  be 
modeled  several  different  ways.  For  example,  a  sonar  model  at  a  facility  with  supercomputing  capabilities  might 
use  a  very  high  fidelity  signal  propagation  model,  while  another  facility  would  run  a  low  fidelity  abstraction  on  a 
laptop  computer.  This  woidd  not  affect  other  aspects  of  the  SEA,  but  may  impact  what  questions  can  be  studied 
at  each  facility. 

To  support  the  S&T  community,  we  not  only  need  easily  distributable  SEAs  and  component  parts,  but  we  need 
scenario  authoring  tools,  model  building,  and  theory  testing  capabilities.  Also  consider  that  S&T  is  inherently 
multi-disciplinary,  so  we  cannot  assume  that  everyone  has  programming  skills  readily  available. 

Of  course,  the  boundaries  between  all  these  domains  are  “soft.”  In  order  to  ask  trade-off  questions  such  as 
“What  are  the  training  implications  of  introducing  this  new  weapon  system?”,  we  must  look  across  two  domains 
-  procurement  and  training.  This,  again,  is  the  power  of  SEA.  In  the  past,  that  question  likely  would  have  been 
answered  by  two  different  communities  using  two  different  systems  resulting  in  data  that  could  not  be  compared 
fairly  in  order  to  make  a  decision.  SEA  would  use  the  same  simulation  components  to  ensure  that  the  analyst 
was  able  to  make  an  “apples-to-apples”  comparison  (see  Section  2.5). 

Finally,  the  list  of  components  that  SEA  must  aggregate  in  order  to  connect  the  inputs  to  the  outputs  includes: 

•  Physics'.  At  variable  levels  of  fidelity,  we  need  models  to  approximate  physics  -  statics,  dynamics, 
friction,  deformable  surfaces  and  materials,  etc.  -  that  can  be  coupled  to  visual  models  for  analysis. 

•  Environmental :  Models  that  represent  weather,  vegetation,  soil  types,  and  other  environmental 
phenomena  will  be  needed  for  many  SEA  analyses.  Physics  and  environmental  models  are  both  a  part  of 
mobility  models  that  are  critical  to  vehicle  analysis  for  question  concerning  how  a  specific  vehicle  will 
perform  under  certain  conditions. 
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•  Platforms  and  Munitions'.  These  are  the  most  obvious  types  of  components  that  SEA  will  use,  as  they 
represent  equipment  that  may  be  the  focus  of  a  test  or  evaluation,  either  in  isolation  or  as  a  part  of  an 
aggregate  system. 

•  Automation'.  Questions  concerning  what  level  of  automation  might  be  achievable,  or  what  the  impact  of 
automation  might  be  in  a  particular  combat  scenario  demand  models  that  accurately  represent  the 
automation  itself. 

•  Interoperability'.  An  essential  ingredient  that  allows  modules  that  may  have  been  developed 
independently  of  one  another  to  function  together.  Common  data  models  are  a  key  means  to 
interoperability  because  they  circumvent  the  use  of  Application  Programmer  Interfaces  (APIs)  which 
can  be  brittle.  This  also  allows  heavy  leverage  of  ongoing  work  in  the  M&S  community. 

•  Artificial  Intelligence  (AI):  We  use  this  term  very  broadly  to  include  all  aspects  of  AI  such  as  Human 
Behavior  Models  (HBM),  Human,  Social  Cultural,  Behavioral  (HSCB)  models,  agent  models, 
production  systems,  intelligent  tutoring  systems,  etc. 

•  Virtual  Humans'.  Any  form  of  human  model  that  could  be  anthropometric,  cognitive,  perceptual, 
physiological,  etc.,  which  is  meant  to  simulate  human  capabilities  for  testing. 

•  Analytics:  For  large  studies,  the  number  of  design  permutations  may  be  in  the  thousands  or  even  higher. 
SEA  requires  ways  to  analyze  and  visualize  the  outputs  so  that  the  analyst  can  make  sense  of  the  data. 

•  VV&A:  From  the  M&S  community,  VV&A  is  Verification,  Validation,  and  Accreditation.  Since  SEA  is 
made  up  of  models,  the  validity  of  those  models  is  key  to  the  success  of  SEA  as  an  approach. 
An  important  factor  here  is  that,  while  we  may  be  able  to  validate  models  A  and  B,  when  we  aggregate 
A+B,  we  may  not  know  if  the  aggregate  model  is  valid  as  well. 

•  LVC:  We  use  the  idea  of  simulation  components  without  stating  what  those  components  can  be. 
They  may  be  simulations  of  individual  assets  or  personnel  controlled  by  individuals  (Virtual).  They  may 
be  software  simulations  of  individual  assets  or  personnel  controlled  by  a  computer  (Constructive), 
or  they  may  be  data  feeds  from  personnel  in  a  live  exercise  using  real  equipment  (Live).  This  is  why 
LVC  is  one  of  the  other  NATO  focus  areas  of  interest  to  SEA  (see  Section  5.1). 

We  do  not  claim  that  this  list  is  complete.  In  fact,  we  are  sure  it  is  not.  It  is  only  meant  to  identify  several 
fundamental  technology  areas  on  which  SEA  depends.  More  will  be  added  as  SEA  continues  to  mature. 

2.5  Use-Cases 

Synthetic  environments  are  used  by  a  number  of  different  groups  for  a  number  of  different  reasons.  The  one 
thing  all  uses  have  in  common  is  that  they  are  intended  to  determine  some  configuration,  apparatus,  policy,  etc., 
that  will  optimize  performance.  In  identifying  who  these  groups  are  and  what  their  needs  are,  we  must  first  look 
at  a  simple  model  of  the  acquisition  process  (see  Figure  5).  While  this  is  an  illustration  taken  from  the  US 
Department  of  Defense,  it  generalizes  functionally  to  international,  industrial,  or  other  applications. 
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Figure  5:  Acquisition  Cycle  Phases  and  Milestones. 


The  key  parts  of  the  cycle  we  are  interested  in  are  at  the  beginning  of  the  process.  While  it  may  appear  that  the 
process  is  linear  and  sequential,  in  practice,  it  functions  two  ways.  The  first  way  is  that  a  concept  is  identified, 
gaps  are  addressed,  and  the  system  is  developed  and  deployed.  In  this  case,  requirements  are  often  provided  by 
the  procurement  process  itself.  However,  the  capabilities  development  community  also  develops  new  concepts 
independent  of  the  procurement  process.  New  concepts  that  may  be  completely  disruptive  to  current  doctrine  or 
policy  must  be  tested  before  being  considered.  The  need  to  explore  a  complex  design  space  is  implied  with  the 
ability  to  test  and  evaluate  plausible  solutions  for  consideration.  This  procurement  process  and  the  many 
different  perspectives  it  entails  motivates  the  need  for  SEA. 

Throughout  the  procurement  process,  SEA  enables  an  expansion  and  pruning  of  design  alternatives.  In  Figure  6, 
the  green  nodes  represent  active  designs  that  are  subsequently  expanded;  the  red  nodes  represent  designs  that 
have  been  discarded  (likely  because  they  were  found  lacking  in  some  way  when  the  assessment  criteria  was 
applied);  and  finally,  the  yellow  node  represents  the  evolved  “best”  design  for  this  need.  Of  course,  it  is  never 
guaranteed  to  be  the  “best”  design,  but  through  the  process,  it  is  guaranteed  to  be  better  than  its  predecessors. 


Figure  6:  Expanding  and  Pruning  of  Design  Considerations  Throughout  the  Process. 
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Table  2:  SEA  User  Groups  and  Potential  Use-Cases. 


User  Groups 

Application  Domain 

Uses 

Customers 

Support  to  Operations 

Doctrine  analysis 

Testing  new  organisational  structures 

Users 

Capability  Development 

Doctrine  validation,  Operational  analysis, 
Operational  requirements  definition, 
Research  and  technology,  Concept 
development  and  experimentation 

Suppliers 

Mission  planning 

Mission  Rehearsal 

Risk  identification 

Coordinators 

Trust  building 

Training  and  Education 

Plan  and  conduct  training 

Advisors 

Procurement 

Design  risk  reduction 

Test  and  Evaluation 

2.5.1  Capability  Development 

Within  capability  development,  one  of  the  most  common  uses  for  SEA  is  in  hypothesis  testing  and  discovery. 
When  a  theory  is  proposed,  often  the  most  plausible  way  to  test  it  is  through  a  simulation,  especially  when  the 
real-world  task  is  too  expensive,  too  dangerous,  too  inaccessible,  or  too  complex  for  practical  use.  However, 
this  immediately  creates  a  natural  friction  between  realism  and  control. 

The  conundrum  lies  in  the  fact  that  real-world  tasks  are  often  too  complex  to  yield  definitive  results  from 
experimentation.  Y et  simplification  of  the  task  for  laboratory  study  is  often  criticized  as  not  real  or  too  sanitized 
to  be  believed.  What  is  a  researcher  to  do?  We  often  create  simulated  environments  that  mimic  real  tasks,  albeit 
at  lower  fidelity.  But  how  does  the  researcher  validate  the  task  set  and  environment,  even  under  limited  realism 
conditions? 

Because  SEA  is  a  collection  of  simulation  components,  each  of  which  models  some  aspect  of  a  situation  of 
concern,  it  offers  experimental  opportunities  not  possible  in  the  real  world  because  we  are  able  to  isolate 
components  for  study.  The  conventional  scientific  method  calls  for  theory  to  be  validated  via  hypothesis  testing 
that  requires  that  a  set  of  dependent  variables  be  measured  against  a  set  of  independent  variables.  In  HSI, 
for  example,  this  is  often  an  empirical  study  with  human  subjects.  What  SEA  facilitates  is  the  use  of  a  modified 
scientific  method  where  simulation  is  used  preceding  live  tests  as  a  more  cost  effective  and  rapid  way  of 
exploring  more  combinations  of  dependent  and  independent  variables.  Then,  we  take  good  options  and  explore 
these  in  more  detail  with  more  expensive  live  tests. 

In  Figure  7,  conventional  experimentation  tests  a  hypothesis  by  measuring  dependent  variables  as  independent 
variables  are  manipulated.  What  simulation  has  enabled  is  the  use  of  simulation  as  the  test  enviromnent.  While 
this  isn’t  unique  to  SEA,  what  is  important  is  that  SEA  allows  a  researcher  to  manipulate  a  single  component 
(for  example,  a  display  interface  or  a  sensor),  measure  the  effect  on  performance,  and  then  iterate. 
More  importantly,  the  iteration  can  be  done  in  more  than  one  laboratory  and  we  can  compare  results  because  the 
SEA  was  the  same  -  the  simulation  components  were  the  same  and  operational  scenarios  were  the  same. 
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Figure  7:  A  Modified  Scientific  Method  Enabled  by  Simulation. 


2.5.2  Procurement  and  Analysis 

One  of  the  key  assumptions  driving  the  requirement  for  SEA  is  that  design  spaces  are  becoming  more  complex 
and  our  ability  to  explore  them  is  not  improving  at  a  reasonable  pace.  The  only  way  this  can  be  addressed  and 
managed  is  if  SEAs  are  rapidly  reconfigurable  and  if  searches  into  large  portions  of  the  design  space  can  be 
automated,  or  semi-automated.  There  is  precedent  for  this  idea:  the  FACT  program  (US  Marine  Corps)  is  an 
online  tool  supporting  acquisition  that  does  automated  assessments  within  certain  parameters  and  reports  the 
“best  fit”  outcomes  that  can  then  be  independently  analyzed.  Other  systems  also  facilitate  some  sort  of  “Monte 
Carlo”  technique  for  pseudo-randomly  exploring  design  alternatives  for  “goodness  of  fit.”  However,  these  are 
systems,  not  architectures,  and  consequently  they  do  not  help  us  when  we  change  domains  or  organisations. 

A  conceptual  solution  (see  Figure  8)  is  to  start  with  an  architectural  framework  (DODAF,  MODAF  or  some 
other  tool  might  be  useful  but  are  certainly  not  the  only  tools  that  fit  this  description)  that  allows  for  functional 
decomposition  of  a  complex  problem  space.  This  is  a  “model”  of  the  problem  with  appropriate  metrics  that  will 
allow  us  to  assess  design  alternatives.  The  functional  decomposition  woidd  then  be  used  to  select  an  appropriate 
set  of  simulation  modules  which,  when  configured,  accurately  represent  the  model.  At  this  point,  the  SEA 
applies  the  metrics  to  each  proposed  solution  and  visualizes  the  results.  Constraints  can  then  be  applied  and  the 
process  iterates  to  produce  more  solutions  for  consideration. 


Architectural  Framework  Simulation  Modules 


Requirements 


Metrics 


Experiment  1 

h 

Experiment  2 

Experiment  n  L 

_> 

T 

Interventions 


MODELING  ARCHITECTURE/COM  POSITION  SIMULATION/EXECUTION 


Figure  8:  Simple  Model  of  SEA  Use  in  a  Conventional  System  Engineering  Process. 
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For  a  more  concrete  example,  consider  the  case  shown  in  Figure  9.  This  is  an  anti-submarine  warfare  scenario. 
Note  that  the  SEA  is  a  set  of  independent  models  (submarine,  towed  array,  SFI-60,  dipping  sonars,  DDG,  sonar 
operator)  that  communicate  through  a  centralized  common  data  model  (1).  The  scenario  (2)  is  a  validated 
scenario  that  allows  for  the  testing  of  performance  under  realistic  operational  conditions. 


Submarine  Model 


SH-60  Model 


Dipping  Sonar  Model 
Dipping  Sonar  Model  B 


Towed  Array  Model 


DDG  Model 


/  \ 


Sonar  Operator  Model 


Display  Design  A 


Display  Design  B 


t 


Figure  9:  A  Detailed  Use-Case  for  SEA  in  an  ASW  Application. 


The  first  question  we  want  to  ask  with  this  SEA  is,  “Which  results  in  a  better  probability  of  detection,  Display 
Design  A  or  Display  Design  B?”  (3).  We  obtain  a  suitable  user  group  and  test  performance  on  measures  related 
to  probability  of  detection  based  on  the  given  scenario.  One  of  the  designs  should  emerge  as  the  better  of  the 
two,  but  it  likely  has  a  price  tag  associated  with  it  so  a  decision  will  have  to  be  made  as  to  whether  or  not  the 
improvement  is  worth  the  cost. 

But  we  could  ask  a  different,  albeit  related  question  that  focuses  on  an  immediate  solution  decision.  There  is  a 
vendor  who  claims  their  dipping  sonar  sensor  is  15%  better  than  the  one  we  are  currently  using.  What  if  we  want 
to  know  “Which  results  in  better  probability  of  detection,  a  new  dipping  sonar  sensor  or  improved  operator 
display?”.  We  need  to  hold  everything  else  constant  and  compare  performance  using  the  same  user  groups 
against  the  same  scenario,  but  now  we  will  test  the  two  sonar  models  (4)  against  the  displays  (3).  The  resulting 
performance  data  will  allow  the  decision-maker  to  fairly  compare  the  improvements  of  each  alternative,  taking 
into  consideration  its  respective  cost.  This  is  what  acquisition  managers  and  analysts  need. 

2.6  SEA  Performance  Metrics 

These  use-cases  illustrate  the  immediate  and  long-term  impact  of  SEA.  Flowever,  we  still  need  to  understand 
how  to  measure  the  added  value  of  SEA  as  compared  to  today’s  best  practices.  A  reasonable  initial  set  of  criteria 
for  SEA  would  include: 
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•  Measurable  reduction  in  cost  and  time  associated  with  the  use  of  simulation  in  assessment; 

•  Measurable  reduction  in  overall  lifecycle  costs; 

•  Direct  linkage  from  SEA  scenarios  to  operational  tasks  and  mission  effectiveness  metrics; 

•  Calibrated  performance  metrics  with  actual  operators; 

•  Distributable  to  a  broad  community  (capability  development,  procurement,  analysis);  and 

•  Variable  fidelity  as  required  by  the  problem  set. 

Not  all  of  these  are  easy  to  measure,  but  they  highlight  the  areas  where  SEA  is  expected  to  have  the  greatest 
impact. 

2.7  Manage  Risk,  Increase  Trust 

At  the  onset  of  a  program,  we  assume  a  design  meets  stated  requirements,  the  technology  gap  is  eliminated 
through  the  use  of  up-to-date  products  and  trust  (or  confidence)  on  the  part  of  program  management  that  the 
product  will  meet  stated  goals  is  high  (see  Figure  10).  As  the  program  moves  into  development  and  deployment 
(Milestone  B  and  beyond  in  Figure  5),  the  design  inevitably  falls  out  of  step  with  reality.  Requirements  change, 
budgets  change,  and  schedules  change,  all  of  which  require  modifications  to  the  program  execution  plan  and 
design.  The  further  along  in  the  cycle,  the  more  expensive  the  change  and  the  lower  the  trust  in  management  that 
the  change  will  not  adversely  impact  the  entire  program  (or  at  least  that  the  impact  is  known).  If  SEA  were  used 
throughout  the  program  lifecycle,  design  trade-offs  could  be  analyzed  constantly.  The  objective  is  to  raise  trust 
in  management  while  lowing  costs  and  consequently  making  it  easier  to  close  the  technology  gap  throughout 
program  execution,  not  just  at  the  onset. 


Tcrtinoto©'  Up, 


Figure  10:  SEA  Closes  Gaps  in  Technology,  Costs,  and  Trust. 
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Furthermore,  this  same  methodology  is  useful  when  capability  development  explores  new,  disruptive  concepts. 
If  capability  development  wants  to  explore  a  concept  that  radically  alters  force  structures  or  the  use  of  limited 
assets,  SEA  will  be  the  way  that  this  will  be  explained  to  decision-makers  and  materiel  developers.  It  will  also  be 
how  measurable  benefits  of  the  new  approach  will  be  evaluated  and  compared  against  competing  solutions. 
This  is  currently  most  often  done  via  presentation  and  notional  data;  consequently,  these  critical  decisions  are 
made  based  on  instinct  rather  than  good  data. 


3.0  CORE  COMPONENTS  AND  REFERENCE  ARCHITECTURE 

How  do  our  initial  ideas  about  what  SEA  is  and  how  it  should  be  implemented  (see  Section  2.4)  match  up  to 
what  our  NATO  members  are  doing?  We’ll  first  identify  common  elements  and  components  and  then  put  the 
parts  together  into  a  reference  architecture  that  can  be  useful  in  thinking  about  an  international  SEA  capability  to 
be  shared  by  all. 

3.1  Common  Components,  Unique  Components 

We  reviewed  each  of  the  summaries  of  SEA  efforts  for  each  of  our  NATO  members  to  identify  common 
elements.  Since  there  currently  are  no  shared  efforts  among  the  members,  we  had  to  allow  for  a  broad  definition 
of  components  in  terms  of  their  function,  not  their  implementations,  which  are  certainly  different.  Table  3  shows 
the  results  of  that  comparison. 


Table  3:  Common  Elements  Among  NATO  Members  SEA  Efforts. 


Core  Elements 

Germany 

Canada 

France 

UK 

Netherlands 

USA 

Multiple  User  Communities 

X 

X 

X 

X 

X 

X 

Human  Models 

X 

X 

X 

X 

X 

Model  Aggregation 

X 

X 

X 

LVC 

X 

X 

X 

X 

External  Data  Sources 

X 

X 

X 

X 

X 

X 

HPC 

X 

Architectural  Framework 

X 

X 

Common  Data  /  Backplane 

X 

X 

Physical  Mock-ups 

X 

X 

X 

X 

Full  Simulation  Integration 

X 

X 

X 

X 

X 

X 

Extensible  Computing 

X 

X 

Visualization  of  Results 

X 

X 

X 

X 

X 

Reusable  Models 

X 

X 

X 
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All  countries,  in  some  fashion,  were  addressing  multiple  user  communities  in  their  efforts.  The  most  common 
“customer”  was  the  acquisition  community,  but  training  was  also  typically  included.  Not  every  Nation  has  as 
robust  of  an  S&T  community  as  the  US,  so  much  of  that  work  was  unique  to  the  US. 

Most  countries  also  had  some  notion  of  a  human  model  -  both  at  the  individual  and  group  levels.  Germany  had 
the  most  sophisticated  work  in  human  modeling  that  we  observed,  but  France  was  very  detailed  in  the  specific 
areas  of  concern  to  their  work  (cognitive  and  perceptual). 

We  considered  a  very  loose  definition  of  model  aggregation,  which  woidd  be  the  use  of  multiple  sub-models  to 
make  a  bigger,  more  complex  model.  Not  every  country  was  using  these  types  of  models. 

LVC  is  another  area  that  is  more  common  than  we  anticipated  it  to  be.  Most  countries  have  some  form  of  LVC 
in  use  within  their  SEA  efforts.  Even  Canada  has  virtual  and  constructive  simulators  in  use  in  their  vVic  system. 

All  countries  facilitate  the  use  of  external  data  sources  in  their  models  and  simulations.  That  data  could  come 
from  any  number  of  sources,  from  static  databases,  to  live  data  feeds,  to  high  performance  computing. 

Only  the  US  highlighted  the  use  of  EIPC  in  the  ERS  initiative,  but  it  was  apparent  that  the  Canada,  Germany  and 
the  UK  are  all  planning  for  EIPC  in  the  future  (if  they  aren’t  doing  so  already). 

The  UK  and  US  are  the  only  countries  currently  using  architectural  frameworks  to  statically  define  systems 
before  they  even  build  the  simulator.  The  UK  and  the  US  are  also  the  only  two  countries  with  plans  or  efforts  to 
synchronize  multiple  models,  simulators,  and  data  sources  via  some  sort  of  common  data  model  or  “backplane” 
as  it  is  called  in  the  AM1E  program  in  the  US. 

Many  efforts  use  physical  mock-ups,  and  Canada  is  a  leader  in  this  area.  All  countries  allow  for  the  integration 
of  full  simulators  in  their  SEA  work.  Full  simulator  integration  is  the  obvious  precursor  to  model  integration. 

Both  Canada  and  France  have  constructed  architectures  that  allow  for  easily  scalable  computing  capabilities. 
They  add  computation  by  adding  servers,  without  the  need  for  system  integration. 

There  was  some  form  of  results  visualization  for  all  countries,  but  clearly  some  are  ahead  of  others.  We’ll  point 
out  the  work  in  Germany  and  the  UK  in  this  area. 

Finally,  there  are  hints  of  reusable  modeling  efforts  everywhere,  but  only  Germany,  the  UK  and  the  US  have 
efforts  underway  that  touch  on  this  capability,  which  most  of  us  feel  is  central  to  the  SEA  concept. 

3.2  A  Reference  Architecture  for  SEA 

Now  that  we  know  what  is  common  to  all  the  efforts  described  in  this  report,  we  can  attempt  to  further  refine  the 
very  coarse  architectural  model  proposed  in  Section  2.4  into  something  that  might  be  implemented. 

Figure  1 1  shows  the  model  we  have  created  based  on  all  the  inputs  we  received  from  RTG  EIFM-216  members. 
We  divided  it  into  six  categories: 

•  Resources  are  the  external  resources  that  are  probably  not  even  meant  specifically  for  SEA  that  can  be 
integrated  as  needed  -  we  include  architectural  frameworks  here  as  well  as  EIPC  and  other  data  sources, 
be  they  static  or  dynamic. 
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•  Models  and  Systems  are  the  individual  systems  (could  be  actual  fielded  hardware),  constructive 
simulators,  live  data  feeds,  virtual  simulators  (either  whole  systems  or  those  built  from  components), 
virtual  humans,  and  cost  models. 

•  Common  Data/Backplane  is  the  very  important  mechanism  for  allowing  models  to  interoperate  at  the 
model  level.  Recall  that  this  is  what  AMIE  is  attempting  to  do.  It  facilitates  reuse  and  aggregation, 
both  of  which  are  essential  features  of  SEA. 

•  Inputs  refers  to  calibrated  scenarios  and  metrics.  Users  need  to  supply  both  of  these.  Scenarios  can  focus 
on  capability  gaps,  novel  concept  exploration,  equipment  testing,  comparative  analysis,  or  any  of  the 
other  objects  we  have  discussed  in  this  report.  Metrics  directly  tie  into  the  outputs  since  they  specify 
what  will  be  captured  and  reported. 

•  Outputs  are  the  results  as  specified  by  the  metric  inputs,  but  also  include  visualization  of  results  for 
sense  making,  and  then  finally  decision  support  tools  to  aid  the  decision-maker  in  understanding  trade¬ 
offs  and  their  implications. 

•  User  Communities  are  the  different  user  groups  that  SEA  is  meant  to  serve.  We  add  these  just  to  remind 
us  that  each  community  has  different  needs  that  SEA  needs  to  accommodate.  SEA  will  not  be  successful 
unless  all  of  these  communities  find  value  in  it. 


Resources  Models  &  Systems 


Inputs 


Outputs 


User  Communities 


Figure  1 1 :  A  Detailed  Architectural  Model  for  SEA. 
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Architectures  are  about  how  the  pieces  fit  together.  For  complex  software  systems  like  SEA,  they  are  about 
protocols,  data  models,  file  formats,  and  service  contracts.  We  made  the  statement  earlier  in  this  report  that  there 
was  very  little  in  SEA  that  was  ground  breaking,  as  a  ne w  technology.  It  is  the  architecture  that  is  new,  which  is 
driven  by  a  novel  way  of  thinking  about  the  use  of  simulation  to  explore  complex  design  spaces  for  a  variety  of 
purposes.  Below  is  a  partial  list  of  existing  technologies  on  which  SEA  depends: 

•  Common  data  models; 

•  Interoperability  protocols; 

•  VV&A; 

•  LVC; 

•  Cost  modelling; 

•  HPC; 

•  Authoring/modding  tools; 

•  Architectural  frameworks; 

•  Data  visualization;  and 

•  Decision  support  tools. 

Each  of  these  is  and  has  been  a  major  worldwide  science  and  technology  investment  for  a  number  of  years. 
SEA  seeks  to  leverage  that  investment  in  new  ways  to  build  simulation  environments  that  have  unprecedented 
flexibility,  agility,  and  robustness. 

3.3  Technical  Requirements 

Making  the  statement  that  SEA  doesn’t  need  to  invent  any  key  technologies  in  order  to  be  successful  does  not 
imply  that  all  of  these  technologies  are  at  the  necessary  level  of  maturity  or  are  ready  to  perform  their  duty  in  an 
SEA  today.  We  list  below  some  of  these  areas  and  our  ideas  for  improvement  needed  in  order  to  make  SEA 
function  the  way  we  envision: 

•  Common  Data  Models :  The  work  done  in  the  M&S  community  is  excellent,  but  SEA  may  require 
dynamic  extensibility  of  models  given  the  breadth  of  problem  spaces  it  will  operate  within. 
For  example,  a  simulation  typically  needs  to  be  aware  of  everything  that  can  possibly  occur  at  runtime. 
What  if  it  could  learn  about  new  assets,  events,  and  capabilities  on  the  fly? 

•  Interoperability  Protocols'.  This  is  fairly  robust  but,  for  example,  the  shortcomings  of  HLA  have  been 
well  documented.  That  is  not  to  say  that  HLA  has  no  place  in  SEA,  but  rather  that  new  protocols 
(also  dynamic  if  possible)  may  be  needed. 

•  VV&A:  We  gave  an  example  earlier  where  a  validated  Model  A  is  aggregated  with  a  validated  Model  B. 
What  do  we  know  about  the  new  Model  A+B?  Is  it  valid? 

•  LVC:  This  is  one  of  the  areas  where  SEA  will  leverage  another  RTG  (HFM-221).  Synchronization 
issues  are  key  to  LVC.  Also,  SEA  allows  for  the  use  of  multi-resolution  models,  from  high  fidelity  to 
low  depending  on  application  and  available  resources.  Live  is  the  ultimate  fidelity  and  should  be 
considered  another  model  in  the  SEA  framework. 

•  Authoring/Modification  Tools:  Many  of  the  analysts  who  need  SEA  do  not  have  programming  skills  yet 
they  need  to  be  able  to  carefully  craft  scenarios  to  suit  their  purposes.  In  gaming,  modding  tools  are 
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becoming  more  sophisticated,  but  in  M&S,  we  are  lagging  behind.  This  is  an  area  ready  for  significant 
improvement. 

•  Architectural  Frameworks'.  These  are  “languages”  that  allow  one  to  fully  describe  an  application  or 
system  in  documentation  before  it  is  prototyped  or  built.  What  we’d  like  is  a  way  to  go  directly  from  the 
description  language  into  an  executable  environment  for  testing.  Some  aspects  of  this  capability  exist 
now. 

•  Data  Visualization'.  As  sophisticated  as  data  visualization  is  today,  there  is  always  room  for 
advancements.  The  types  of  data  we  expect  to  result  from  SEA  explorations  of  complex  design  spaces 
are  unknown.  We  don’t  know  what  they  look  like  or  how  we  will  view  them  to  find  the  best  candidate 
designs. 

•  Decision  Support  Tools:  Once  we  have  ways  to  look  at  the  data  to  help  us  understand  it,  we  next  need 
these  to  be  integrated  into  tools  that  help  the  analyst  make  a  decision  that  could  involve  a  very  large 
investment.  “What  if’  games  might  be  useful  here  to  allow  the  analyst  to  play  with  the  variables, 
possibly  including  multiple  forms  of  sensitivity  analysis. 


4.0  RTG  HFM-216  MEMBER  SEA  ACTIVITIES 

In  this  section,  we  will  briefly  describe  the  SEA  activities  ongoing  in  each  of  the  RTG  HFM-216  Member 
Nations.  The  discussion  will  focus  on  application  domains,  architecture,  and  possible  relationships  to  other 
projects,  either  within  that  Nation  or  with  other  partners. 

4.1  Canada 

Defence  Research  and  Development  Canada  discussed  the  Virtual  VICTORIA  (vVic)  project  and  the  Maritime 
Capability  Evaluation  Laboratory  (MCEL).  This  is  an  interesting  case  for  SEA  because  the  objectives  of  the 
program  are  very  SEA-related,  but  they  are  currently  leveraging  extensive  use  of  physical  mock-ups  and 
prototypes.  They  see  an  increased  use  of  simulations  for  assessment,  but  it  would  likely  be  integrated  into  their 
efforts  with  physical  prototypes  given  their  investment  in  this  area. 

The  vVic  project  uses  a  plywood  submarine  mock-up  fitted  with  video  games  to  assess  user  interface  options. 
In  Figure  12,  the  physical  mock-up  shows  a  series  of  consoles  each  running  a  version  of  the  “Dangerous  Waters” 
video  game1.  In  the  bigger  picture,  the  MCEL  is  intended  to  include  two  reconfigurable  spaces  for  vVic-types  of 
projects.  All  of  this  work  is  for  measuring  team  performance. 


1  http://www.sonalystscombatsims.com/dangerous_waters/. 
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Figure  12:  The  Physical  Mock-Ups  Used  in  vVic. 


Architecturally,  they  use  virtual  machines  and  off-the-shelf  workstations  for  their  hardware  to  reduce  costs. 
The  schematic  in  Figure  13  shows  that  any  set  of  devices  (which  could  be  anything  from  a  video  game 
workstation  to  a  handheld  device)  can  be  connected  to  the  network  and  be  driven  by  their  compute  servers  that 
run  specific  models  as  needed.  This  approach  simplifies  their  configuration  management  but  does  present  some 
limitations  in  what  they  can  exercise.  They  archive  all  data  and  can  scale  to  any  size  event  just  be  adding  more 
devices  or  servers. 


Figure  13:  A  Simple  Schematic  of  the  Key  Components  Planned  for  MCEL. 
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The  uses  cases  for  MCEL  are  exactly  what  SEA  is  meant  to  address  -  testing  system  interoperability, 
integration,  perfonnance,  demonstrate  new  technology,  measure  operator  performance  and  team  performance, 
and  test  the  experimentation  process.  It  allows  them  to  “go  to  sea”  before  they  go  to  sea. 

4.2  France 

The  French  Armed  Forces  Biomedical  Research  Institute  (1RBA)  has  a  number  of  human  factors  projects 
utilizing  synthetic  environments.  Some  are  interested  in  perception  (sensor  vision,  3D  vision,  3D  sound,  multi- 
sensory  perception),  while  others  are  using  Force  simulations  to  focus  on  psycho-ergonomics  dimensions  (stress 
management,  mindfulness,  crew  work).  A  basic  concept  behind  SEA,  lowered  costs  through  modular  design, 
would  be  helpful  in  designing  future  studies  in  a  controlled  and  realistic  synthetic  environment. 

With  the  quickening  evolution  of  technology,  the  needs  for  military  simulation  in  training  have  changed.  There 
has  been  a  shift  from  models  to  drive  complex  systems  (aircraft,  ship,  etc.)  to  using  tactical  training  in  distributed 
simulated  environments.  This  evolution  has  been  mainly  driven  by  the  need  of  the  French  Army  (Armee  de 
Terre)  who,  for  most  training  situations,  do  not  need  the  same  high-cost  simulators  and  high  degree  of  realism 
required  by  the  Air  Force  (Armee  de  l’Air).  The  Army  wants  low-cost  devices  to  train  basic  tactical  team  skills. 
These  low-cost  simulators  change  the  economic  policy  of  simulation  and  push  to  industrial  partners  to  optimize 
the  techniques  and  dependences  to  fit  the  Force’s  need.  Today,  distributed  simulation  has  expanded  thinking  in 
terms  of  autonomous  simulation  capabilities  with  modular  architecture,  and  highlights  the  problem  of  data 
governance.  Compared  with  other  Forces,  the  French  Army  is  well  ahead  in  its  use  of  distributed  simulation  with 
reusable  modules.  This  evolution  to  a  tactical  level  of  simulation  with  many  different  entities  permits  training 
many  users  in  large  exercises,  utilizing  multiple  sites  with  multiple  platforms  in  a  joint  or  multi-national 
exercise.  These  exercises  can  use  real  and  simulated  entities  and  LA.  Simulations  no  longer  stand  alone. 
The  French  Army  seeks  to  develop  centralized,  basic  shared  data  for  simulations  and  to  expand  that  into  an 
architecture  for  all  defence  simulations,  both  internal  and  shared  with  Allies. 

The  evolution  of  simulation  in  French  Defence  was  assisted  by  ADIS,  a  group  formed  in  1994.  ADIS  is  a  French 
corporate  organisation  gathering  the  Armed  Forces,  DGA  (the  MoD  Procurement  Agency)  and  the  Defence 
industry  in  order  to  promote,  explain  and  facilitate  the  use  of  simulation  for  the  benefit  of  the  French  Defence 
community.  This  group  is  governed  according  to  an  organisational  charter  approved  by  the  DGA  and  the  Armed 
Forces.  The  goal  is  also  to  improve  the  interoperability  of  simulation  applications  and  tools,  as  well  as  the  reuse 
of  simulation  components. 

The  global  factors  that  drive  architecture  of  simulation  in  French  Defence  have  to  be  taken  into  account  at  the 
level  of  basic  synthetic  environment  questions.  In  design,  using  simulation  (synthetic  environment)  is  used 
extensively  in  industry.  For  HMI  design  in  defence,  DGA  has  developed  platforms  to  help  operational  users 
define  the  technology  they  want  to  use  tomorrow.  IBEO  (Illustrateur  de  Besoin  d’Exploitation  Operationnelle) 
are  platforms  that  are  developed  to  specify  and  assess  new  tools,  new  HMI  and  new  work  organisation.  These 
platforms  are  a  part  of  a  more  global  human  centered  design  approach  performed  at  the  DGA. 

For  example,  the  process  applied  to  the  IBEO  “Military  Integrated  Bridge  System”  for  the  French  Navy  was  as 
follows: 

•  Goal:  The  Navy  needs  to  reduce  the  number  of  people  on  warships. 

•  First  Step:  Analyze  the  present  situation  and  define  human-machine  function  allocation. 

•  Second  Step:  Conduct  task  modeling  to  define  work  organisation  and  user  roles. 

•  Third  Step:  Simulate  different  tasks,  devices,  and  work  organisation  in  a  realistic  scenario. 
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Figure  14:  An  Example  of  this  Process  Applied  to  Ship  Design. 

(Adapted  from  Pellen-Blin,  Bry  and  Chouvy,  DGA) 

This  method  is  based  on  user-centered  design  and  is  intended  to  be  iterative  and  assess  solutions  in  realistic 
scenarios.  Assessment  is  based  on  system  performance,  user  collaboration,  communications,  attention,  workload, 
or  whatever  is  appropriate  to  the  scenario.  Moreover,  this  methodology  can  facilitate  new  ideas  as  they  emerge. 

Whether  the  focus  is  on  research,  training  or  design,  assessment  of  the  use  of  synthetic  environments  is  needed 
to  adapt  simulation  to  the  human  and  improve  performance. 

4.3  Germany 

Fraunhofer  FKIF.  has  a  number  of  related  projects  on  the  use  of  modeling  and  simulation  for  human-systems 
integration  that  are  very  closely  aligned  with  SEA  concepts.  There  is  a  specific  focus  on  soldier  modernization 
and  core  support  for  the  warfighter.  Topics  related  to  readiness,  C41,  mobility,  survivability,  and  sustainability 
are  of  interest.  They  use  a  spectrum  of  simulation  products  ranging  from  low  realism,  high  variability 
(like  constructive  simulations)  to  high  realism  and  low  variability  (like  live  exercises)  to  address  these  topics. 

Key  modeling  and  simulation  topics  in  support  of  these  efforts  include  virtual  human  modeling,  information 
visualization,  game  technologies,  and  authoring  tools.  A  notional  architecture  is  presented  in  Figure  15. 
The  digital  human  can  be  considered  the  centerpiece  since  this  is  about  HSI.  It  needs  to  be  capable  of  supporting 
ergonomic  as  well  as  performance  studies.  Weapons,  armor,  and  other  equipment  are  modeled  and  integrated 
with  the  digital  human  as  required  by  specific  scenarios.  Ergonomic  studies  might  include  interior  designs  of 
candidate  vehicles  for  testing.  They  include  extensive  use  of  constructive  simulations  and  SAFs  for  simulation 
support  for  their  scenarios.  Finally,  they  use  visualization  as  a  key  part  of  understanding  the  massive  data  that 
their  studies  generate  in  order  to  make  effective  decisions. 
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Figure  15:  A  Simple  Schematic  Diagram  of  the  Fraunhofer  Efforts. 


Germany  uses  simulation  as  far  as  they  can,  but  relies  on  field  testing  for  final  evaluation.  This  is  similar  to  what 
we  saw  with  most  member  countries.  They  also  see  agent  technology  and  human  behavior  modeling  as  a 
fundamental  enabler  to  how  they  will  use  simulation  in  the  future  for  SEA  like  activities. 


4.4  Netherlands 

There  are  several  SEA-related  efforts  underway  in  the  Netherlands.  One  of  these  involves  the  use  of  simulation 
in  the  development  and  maturation  of  complex  concepts.  These  types  of  activities  overlap  well  with  the  domains 
outlined  earlier  for  SEA  -  specifically  acquisition  as  in  assessing  new  weapons  concepts,  doctrine  analysis  as  in 
assessing  the  effectiveness  of  a  new  missile  define  approach,  and  organisational  analysis  as  in  assessing  the 
effectiveness  of  command  leadership  organisation.  In  all  of  these  cases,  they  use  these  techniques  to  experiment, 
assess,  and  to  iterate  their  ideas  towards  a  more  mature  concept  that  can  be  developed  and  deployed. 

They  seek  to  meld  the  operational  and  scientific  communities  in  a  shared  virtual  workspace  that  creates  a  vision, 
provides  a  framework  for  experimentation  and  subsequent  exchange  of  ideas,  and  then  revision  of  ideas. 
The  solution  is  not  purely  technological,  however.  They  use  a  mix  of  conventional  workshops  for  brainstorming 
and  idea  generation,  but  then  they  also  use  table-top  gaming  and  simulations  to  experience  the  concept  in  order 
to  understand  its  capabilities  and  limitations. 
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Figure  16:  Table-Top  Gaming  and  Exercises. 


Concept  Development  Implementation 

&  Experimentation  Phase  Phase 


Figure  17:  A  Model  for  Capability  Development. 
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4.5  United  Kingdom 

Cassidian,  a  subsidiary  of  EADS,  has  a  number  of  efforts  that  are  related  to  SEA.  In  particular,  they  have  done 
extensive  work  in  architecture  to  support  SEA  activities  that  we  can  leverage  as  we  think  about  how  SEA  can 
evolve  into  an  international  framework  supporting  the  use  of  simulation  for  assessment. 

Their  process  has  four  key  stages  (see  Figure  18).  They  first  use  static  modeling  techniques  as  a  way  to  capture 
requirements  and  communicate  with  the  customer  so  there  is  a  clear  understanding  before  they  proceed. 
This  also  involves  the  use  of  case  studies  and  conceptual  frameworks.  The  dynamic  model  is  able  to  model  the 
interactions  between  independent  components  as  well  as  the  influence  of  one  part  on  another.  They  use  system 
dynamics  modeling  tools,  among  other  techniques,  to  capture  this.  Before  they  move  on  to  full-fidelity 
simulation,  they  first  use  a  simulation  emulation  phase  that  does  not  involve  users  but  emulates  them  in  some 
way.  It  could  be  considered  a  coarse  evaluation  that  is  meant  to  assess  risk  as  it  relates  to  system  capability  so 
that  design  decisions  can  be  made  earlier.  Figure  19  shows  a  simulation  capability  at  the  emulation  phase  that 
models  interactions  both  symbolically  as  well  as  literally  (in  3D).  Finally,  real  system  simulation  tries  to  model 
the  lull  system  at  whatever  level  of  fidelity  is  needed  to  make  design  decisions  for  the  final  product. 


Sifflul4tic<i  Emu  at  on 


No  jseri  |  yep 


risk 


System 
Capaa  il| 


5r“oblfc<i  Real 
Systems 


toarm-ttie-toop 


Raul  systems 


Hgh  fcfeHy 


Figure  18:  Simple  Architectural  View  Used  by  Cassidian. 
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Figure  19:  Human  Behavior  Models  Used  for  Operational  Planning. 


4.6  United  States 

The  United  States  has  a  number  of  large  efforts  in  the  area  of  SEA.  For  many  years,  individual  programs  have 
seen  the  value  in  using  simulation  to  study  designs  and  concepts  as  they  relate  to  the  overall  performance  of  a 
complex  system.  Until  very  recently,  there  was  little  acknowledgement  that  the  common  goals  among  these 
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programs  could  be  used  to  develop  new  tools  and  capabilities  specific  to  SEA.  In  fact,  as  we  pull  the  lens  back 
further,  this  is  a  goal  of  RTG  HFM-216  for  the  international  community. 

The  Fleet  Integrated  Synthetic  Training  Facility  (FISTFAC)  program,  (COMPACFLT,  sponsored  by  the  Office 
of  Naval  Research)  seeks  to  develop  an  integrated  training,  test  and  evaluation  system  for  a  variety  of  naval 
mission  sets.  It  is  meant  to  be  an  end-to-end  testing  and  training  capability.  The  primary  facility  maximizes 
fidelity  via  large  reconfigurable  spaces  that  allow  different  simulators  or  deployable  systems  to  be  integrated  as 
required  by  a  training  or  evaluation  scenario.  This  capability  enables  some  of  the  most  complex  command-and- 
control  problems  to  be  assessed,  and  experiments  involving  new  equipment  or  procedures  to  be  executed. 
Current  focus  is  on  anti-submarine  warfare,  but  the  facility  is  by  no  means  limited  to  that  domain.  The  schematic 
below  shows  a  configuration  typical  of  this  facility.  Not  only  does  it  integrate  multiple  users,  but  also  multiple 
systems.  Because  each  of  the  parts  are  independent,  variations  of  this  facility  can  be  replicated  elsewhere  at 
differing  levels  of  fidelity. 


Figure  20:  The  FIST2FAC  Schematic  Showing  Various 
Simulators  and  Operators  on  a  Shared  Network. 


Another  related  effort  in  the  United  States  is  Engineered  Resilient  Systems  (ERS).  ERS  is  meant  to  address 
problems  associated  with  changing  requirements  over  the  full  lifecycle  of  a  program.  A  resilient  system  is 
defined  as  one  that  is  effective  over  a  wide  range  of  situations  and  that  is  readily  adaptable  through 
reconfiguration  or  replacement  with  graceful  and  detectable  degradation  of  function.  The  goals  of  ERS  involve 
detailed  trade-off  analyses  across  multiple  mission  contexts  that  can  be  fed  into  requirements  generation  and 
refinement.  Also,  the  ability  to  consider  a  much  wider  range  of  options  is  desired.  While  ERS  itself  is  not 
specific  about  the  use  of  simulation,  clearly,  simulation  will  play  a  large  role.  A  current  effort  at  ERDC  is 
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developing  an  architecture  capable  of  integrating  simulation,  life  systems,  and  people  for  the  purposes  of 
exploring  and  assessing  alternative  options. 
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Figure  21:  ERS  Architectural  Plan  (from  ERDC). 


The  Architecture  Management  Integration  Environment  (AMIE)  being  developed  by  the  Naval  Air  Systems 
Command  is  an  open  architecture  to  support  SEA-like  systems.  With  the  increasing  importance  of  major  joint 
programs  that  must  share  systems  and  models  of  systems  among  different  stakeholders,  an  architecture  is  needed 
that  is  capable  of  unifying  these  into  a  coherent  system  for  evaluation.  AMIE  is  largely  an  interface  standard  that 
enables  components  to  communicate  and  synchronize. 

Figure  22  shows  how  AMIE  is  meant  to  unify  models  developed  for  a  wide  variety  of  purposes  by  a  wide  variety 
of  developers.  These  are  adaptable  and  extensible  in  their  own  right.  They  are  aggregated  via  a  standardized 
“backplane”  that  forces  a  common  format  so  that  the  analyst  can  author  a  test  and  evaluation  simulation  by 
“snapping”  components  together  as  needed.  This  relies  heavily  on  common  data  standards  (many  of  which  are 
reaching  maturity  now)  and  protocols.  The  backplane  must  be  an  open,  non-proprietary  interface.  AMIE  reduces 
redundant  effort  and  increases  reliability  through  reuse.  It  also  forces  the  government  to  be  the  lead  integrator 
and  architect  which  is  entirely  appropriate  (and  needed),  but  also  may  demand  skill  sets  the  government  does  not 
necessarily  have  in  abundance. 
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Figure  22:  Schematic  Diagram  for  AMIE  Showing  Different  Input  Models  Unified  on  the  Backplane. 


Finally,  Figure  23  shows  how  these  efforts  relate  to  each  other.  SEA  is  unique  to  complex  systems  with  human 
operators.  The  introduction  or  inclusion  of  humans  into  any  analysis  brings  with  it  a  number  of  issues  that  may 
not  be  present  in  other  problem  spaces.  ERS  can  be  viewed  as  a  wider  scoped  initiative  focused  more  on  the 
acquisition  cycle.  The  intersection  of  SEA  and  ERS  is  where  decision  support  capabilities  are  developed  that 
facilitate  the  exploration  of  design  options  related  to  overall  system  performance,  systems  and  personnel. 
AMIE  is  mainly  a  way  to  put  the  pieces  together.  It  is  a  key  enabler.  The  intersection  of  AMIE  and  SEA  is 
where  we  develop  software  architectures  for  LVC  and  man-in-the-loop  simulations  that  also  integrate  complex 
models  for  analysis.  The  intersection  of  AMIE  and  ERS  is  similar  but  with  a  broader  scope,  looking  at  very  large 
design  spaces  that  must  include  automated  design  of  experiments,  visualizations  for  sense  making,  and  other 
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analyst  support  tools.  Finally  the  intersection  of  all  of  these  is  a  component  level  architecture  for  interoperable 
simulation  that  enables  plug-in  models  of  system  modules  (which  can  be  people,  equipment,  or  process).  It  is 
tailored  to  the  acquisition  cycle  and  includes  cost  models,  synthetic  test  and  evaluation,  and  iterative  design 
refinement. 


Figure  23:  The  Intersecting  Domains  of  SEA,  ERS,  and  AMIE. 


5.0  RELATIONSHIP  TO  OTHER  NATO  STO  ACTIVITIES 

5.1  Live-Virtual-Constructive  Training  (RTG  HFM-221) 

5.1.1  LVC  Agenda  and  Objectives 

LVC  is  becoming  pervasive  in  the  international  training  and  mission  rehearsal  community  due  to  the  maturity  of 
enabling  technologies  and  the  push  to  drive  costs  down.  However,  RTG  HFM-221  recognizes  that  there  is  still 
little  agreement  as  to  what  constitutes  L,  V,  and  C  or  what  the  aggregate  LVC  means  in  terms  of  training  and 
mission  rehearsal.  The  lack  of  agreement  in  definition  as  well  as  tools  for  interoperability  and  synchronization 
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are  also  barriers.  A  key  objective  of  RTG  HFM-221  is  to  seek  common  goals  and  opportunities  across  NATO 
partners  that  will  enable  joint  exercises  and  training  as  well  as  the  sharing  of  technologies  where  appropriate. 

A  current  focus  is  on  what  it  will  take  to  develop  a  continuous  learning  (or  training)  environment  that  integrates 
LVC  as  a  seamless  and  persistent  part  of  all  training.  If  it  can  be  shown  that  learning  can  be  accelerated  in  this 
manner,  then  the  international  community  can  leverage  this  in  many  ways.  Other  objectives  include  identifying 
best  practices  for  LVC  across  Member  Nations,  methods  of  measuring  performance  in  LVC  environments, 
exploring  the  differences  of  LVC  training  as  compared  to  other  methods,  and  identification  of  candidate 
exemplars  that  can  be  shared  among  members  and  could  be  expanded  into  an  RTG  LVC  demonstration. 

5.1.2  LVC  +  SEA  Opportunities 

We  have  mentioned  a  number  of  areas  where  LVC  fits  into  the  SEA  ecosystem  -  specifically  in  terms  of 
thinking  about  LVC  “components”  as  any  other  component  that  can  be  aggregated  within  SEA  for  a  specific 
assessment.  For  example,  if  we  wanted  to  use  SEA  to  study  the  trade-offs  of  a  new  SOP  for  ASW  scenarios, 
we  might  easily  include  live  operators  on  live  equipment  possibly  in  a  live  exercise  as  part  of  that  assessment, 
even  though  other  parts  were  purely  simulated.  This  will  require  a  full  integration  of  LVC  standards  as  part  of 
the  SEA  architecture. 

5.2  Human  Behavior  Modeling  (HBM)  (MSG-107) 

5.2.1  HBM  Agenda  and  Objectives 

HBM  is  looking  at  modeling  and  simulation  technologies  that  model  humans  at  the  individual  and  small  team 
levels.  There  is  an  immediate  tie  to  LVC  in  that  HBM  seeks  to  solve  problems  associated  with  the  use  of  HBM 
models  in  exercises  that  include  live  players.  They  are  looking  at  a  wide  variety  of  warfighting  domains, 
from  ground  force  applications,  to  air,  and  multi-service  domains.  Much  of  the  work  of  HBM  will  deal  with 
standards  development  that  will  allow  them  to  utilize  the  many  investments  in  HBM  worldwide  that  currently 
cannot  be  unified  into  a  single  system.  This  same  motivation  for  reusability  of  models  is  common  to  SEA. 
This  is  likely  an  area  for  collaboration. 

The  main  objective  of  HBM  is  to  develop  a  reference  architecture  for  HBM  that  can  be  used  in  any  military 
application.  They  will  begin  with  conceptual  modeling  that  crosses  cognition,  decision-making,  planning, 
and  emotions.  The  plan  calls  for  investigation  of  the  architecture  at  the  sub-level  first  working  up  to  an 
overarching  architecture  that  might  be  capable  of  unifying  all  sorts  of  models  that  already  exist  today.  Finally, 
HBM  will  make  recommendations  as  to  the  use  of  the  architecture  in  practical  applications. 

5.2.2  HBM  +  SEA  Opportunities 

HBM  is  but  one  of  the  groups  in  MSG  working  in  areas  of  interest  to  SEA.  There  are  certainly  more.  SEA  needs 
to  pay  attention  to  the  reference  architecture  developed  by  HBM  because  it  must  be  compatible  with  what  we 
develop  for  SEA  since  these  HBM  models  are  going  to  be  common  candidate  models  for  use  in  SEA 
assessments.  In  all  likelihood,  the  HBM  architecture  may  need  to  be  more  general  than  we  need  to  be  in  SEA, 
which  implies  that  letting  HBM  get  out  ahead  of  SEA  may  be  useful. 
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6.0  SUMMARY  AND  RECOMMENDATIONS 

6.1  Summary 

Not  only  does  SEA  provide  a  workable  way  to  solve  the  practical  problems  involving  the  use  of  simulation  in 
the  capability  development  and  procurement  processes,  but  it  creates  opportunities  we  have  never  had  before. 
Calibrated  scenarios  from  one  lab  can  be  used  in  another,  resulting  in  data  that  can  be  fairly  compared. 
Researchers  can  have  realistic  test  environments  available  to  test  and  compare  theories  of  human  performance  in 
a  variety  of  disciplines  without  having  to  expend  precious  resources  learning  the  domain  or  building  “throw 
away”  simulations.  Both  researchers  and  acquisition  professionals  can  explore  a  larger  number  of  potential 
solutions  to  hard  problems  using  trade-off  techniques  for  comparison.  Most  importantly,  capability  development 
and  the  rest  of  the  procurement  community  can  use  SEA  as  a  communication  mechanism.  If  capability 
development  proposes  that  a  new  technology  for  the  use  of  unmanned  vehicles  might  have  a  specific  benefit, 
they  could  communicate  this  to  acquisition  through  realistic  combat  scenarios  within  SEA. 

We  defined  SEA  as  a  modeling  and  simulation  approach  to  experimentation  with  models  and  operators  that 
measures  system  perfonnance  under  “interesting”  perturbations.  SEA  is  a  new  way  to  think  about  the  use  of 
simulation  for  conducting  trade-off  analyses,  and  for  exploring  very  complex  design  spaces.  SEA  is  how  we  will 
break  out  of  the  rut  of  sustaining  improvements  that  we  all  invest  in  every  day  in  favor  of  disruptive  innovation  - 
radically  new  ideas  that  change  the  rules  of  warfare  and  national  defence.  This  is  the  goal  of  SEA  at  its  highest 
level. 

This  report  presented  a  conceptual  model  of  SEA  that  we  used  to  identify  the  key  issues  of  concern.  Use-cases 
were  provided  in  capability  development  and  procurement  to  show  how  SEA  might  be  used  with  significant 
benefit  to  its  user  groups.  We  then  described  the  SEA  activities  ongoing  in  all  our  Member  Nations,  where  there 
are  many  similarities  but  also  many  differences.  We  found  that  SEA,  in  form  if  not  in  name,  is  everywhere. 
All  of  our  member  countries  had  many  of  the  same  ideas  independent  of  each  other  before  RTG  EIFM-216  ever 
began.  What  was  missing  was  a  unified  attempt  to  bridge  our  efforts  in  order  to  share  experiences,  technologies, 
and  results.  This  is  where  RTG  EIFM-216  needs  to  go  next. 

We  used  the  descriptions  of  ongoing  SEA  activities  to  derive  a  detailed  architecture  for  SEA  that  identifies 
resources,  models  and  systems,  input,  outputs,  and  user  communities.  Finally,  we  listed  each  of  the  core 
technology  areas  within  SEA  and  the  unique  technical  barriers  we  need  to  cross  to  realize  SEA  in  its  fullest 
form. 

6.2  Recommendations 

What  is  needed  to  realize  SEA?  We  have  developed  prototype  software  environments  to  test  out  ideas  of 
component-based  architectures.  The  system  engineering  community  has  well-developed  techniques  for 
functional  decomposition  and  task  modeling  that  can  be  used  here.  The  modeling  and  simulation  community  has 
been  working  on  semantic  interoperability  through  common  data  models.  So  many  of  the  pieces  are  either  in 
place  or  are  being  developed  currently.  Elowever,  SEA  still  needs: 

•  A  core  set  of  simple  rules  for  modular  interoperability  that  allow  easily  swappable  parts  to  be 
aggregated; 

•  Common  data  models  that  support  semantic  interoperability  (borrow  from  M&S); 

•  Scenario  authoring  and  modification  tools  to  quickly  adapt  applications  to  test  new  ideas; 
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•  Metric  specification  (and  authoring)  to  ensure  that  the  right  measures  are  applied  and  the  correct  data 
captured  to  test  any  idea; 

•  Visualizations  to  help  analysts  navigate  complex  design  spaces  and  understand  complex  multi-variate 
output;  and 

•  LVC  (live-virtual-constructive)  capability  to  incorporate  live  and  virtual  simulations  into  SEA  test 
packages. 

Some  of  these  needs  may  be  met  by  technologies  already  developed  for  the  training  community  that  provide 
simulation  of  realistic  integrated  missions.  Still  other  advances  in  the  virtual  world  community  offer  promise  for 
needed  realism  and  complexity. 

The  next  step  should  be  an  international  demonstration.  Given  the  common  interests  with  EIBM  and  LVC, 
it  probably  should  be  a  collaborative  effort  since  many  of  the  same  objectives  are  shared  among  all  three  groups. 
There  was  immediate  recognition  among  RTG  HFM-216  members  of  the  redundant  work  each  of  us  appears  to 
be  doing  regarding  SEA  because  there  is  no  architecture  to  allow  us  to  share  models  and  components.  From  a 
practical  standpoint,  the  ideas  behind  SEA  resonate  with  the  community.  We  also  need  to  monitor  the 
technology  “wish  list”  given  in  Section  3.3  and  summarized  earlier  in  this  section.  These  items  require 
significant  resources  to  accomplish,  but  they  are  common  desires  among  all  our  Member  Nations.  As  progress  is 
made,  a  clearing-house  (or  reference  repository)  for  these  technologies  needs  to  be  developed,  so  they  can  be 
shared  and  integrated  into  international  demonstrations  that  show  the  power  of  SEA. 

The  promise  of  SEA  is  great.  The  technical  risks  to  realize  SEA  are  small.  In  fact,  most  components  have  been 
developed  and  demonstrated  in  some  form  already.  SEA-like  projects  are  ongoing  worldwide  without  the  benefit 
of  standards  and  cooperative  interoperability  that  SEA  will  offer.  The  ability  of  synthetic  enviromnents  to 
positively  impact  how  we  discover  and  develop  new  technologies  for  supporting  human  operators  is  too  good  to 
pass  up. 
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